The rapid detection of security threats is critical for organizations to prevent the compromise of their computer systems, data, networks and applications Organizations, whether commercial, educational or governmental, and other enterprises store and transfer the majority of their data in digital form in computer systems and databases. Much of this data is valuable confidential commercial information or private information about individual employees or members that is not intended for public view, and any exposure or manipulation of this data could cause the organization and individuals great financial or reputational damage. Organizations are consistently challenged by threats aimed at stealing, exposing or manipulating this digital data. A large number of these attacks, as reported by the news media, have involved fraud, data breach, intellectual property theft or national security. Some attackers who may have been backed by nation states or organizations with political agendas have taken to more sinister attacks aimed at gaining control or damaging critical infrastructures.
Organizations typically employ a multi-layered network topology to separate various components of their IT infrastructure from the Internet. Workstations and servers are generally protected from direct access via the Internet or other external networks by a web proxy server; Internet traffic is typically terminated at “demilitarized network zones” (DMZ); and the incoming traffic is filtered through a firewall. External attackers normally attempt to penetrate an organization's defenses that are set up at the organization's network perimeter, and many security solutions exist to address external attacks. However, once external attackers breach the perimeter and get onto the internal network, they typically operate under the guise of an internal user by either hijacking an existing user's account or by creating a new user. Internal attackers are more insidious and more difficult to defend against because they are users of the organization's computer network systems. They have legitimate IT accounts, and their unauthorized or illicit activities may generally fall within their areas of responsibility, but exceed what is normal behavior. Attacks may even involve a nexus between external and internal attackers. For instance, detecting illicit activity by an insider customer service representative such as granting a customer an inappropriately large refund may be very difficult to detect.
Most security solutions primarily utilize signatures of known attacks to identify and alert on similar attacks. In order to define signatures for any new threat, the underlying components of the associated threat vector must be studied in detail and signatures of these threat vectors must be made available to a threat detection system. There are several major shortcomings of these signature-based threat detection approaches. The development of signatures for new threats requires an in-depth analysis on an infected system, which is time consuming and resource intensive, and may be too slow to address quickly evolving threats. Signatures do not adapt themselves to changes in threat vectors. Moreover, signature-based approaches are ineffective for zero-day attacks that exploit previously unknown vulnerabilities, and are not available for detecting insider threats originating from within an organization. Identifying insider attacks typically involves constructing various profiles for the normal behaviors of insiders, detecting anomalous deviations from these profiles, and estimating, the threat risk of these anomalies. However, constructing profiles that accurately characterize normal insider behavior is difficult and is not an exact art. For example, many profiles are constructed using statistical approaches for observables that are assumed incorrectly to be normally distributed when they are not. Using such profiles for detecting behavior anomalies can produce erroneous results and lead to many false positive alerts that overwhelm security analysts. Balancing between the risk of missing an actual threat, by using high confidence levels for detection to minimize false positives and using an overly permissive approach that floods security analysts with alerts is a difficult trade-off.
There is a need for systems and methods that address these and other anomaly detection problems in protecting organizations from data breaches and other losses. In particular, there is a need for proactive, reliable adaptive defense capabilities for detecting anomalous activity within an organization's IT infrastructure to identify threats while minimizing false positive alerts. It is to these ends that this invention is directed.